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Experimental  results  of  the  opto-electrical  image 
acquisition  and  processing  unit  with  PAL  optics 


This  document  contains  the  experimental  results  measured  on  the  opto-electrical  image 
acquisition  and  processing  system  with  PAL  optics.  The  main  objective  was  to  provide  a  fast 
image  acquisition  and  processing  system  to  solve  360  degrees  detection  and  tracking  of  moving 
objects. 

Introduction 

The  main  targets  of  this  project  were 

1.  to  specify  and  design  a  standalone  image  acquiring  and  processing  system  with  PAL 
optics,  and 

2.  to  create,  test,  and  evaluate  some  image  processing  algorithms  for  the  images  acquired 
via  the  PAL  lens  of  the  system  mainly  solving  360  degrees  detection  of  moving 
objects. 

The  image  acquiring  and  processing  system  was  specified,  designed,  and  fabricated.  The 
full  hardware-software  environment  of  the  visual  computation  tool  was  successfully  tested.  The 
relay  lens  system  of  the  PAL  optics  has  also  been  specified,  designed,  and  fabricated.  Some  image 
processing  algorithms  were  evaluated  mainly  focusing  on  PAL  lens  images. 


The  standalone  image  acquiring  and  processing  system 

The  high  performance  standalone  unit  (also  called  ACE  box)  is  the  computational 
environment  of  the  64x64  analogic  chip.  The  system  is  implemented  on  PC-104plus  form  factor 
boards.  In  this  way,  standard  industrial  quality  PC-104plus  PC  hosts  a  specially  designed  DSP 
module,  which  drives  the  platform  board  carrying  the  64x64  analogic  chip.  To  reach  high-speed 
communication,  PCI  bus  is  used  as  a  communication  channel  between  the  host  and  the  DSP 
module.  The  DSP  module,  the  platform,  and  the  hosting  industrial  PC  constitute  a  rugged,  high 
performance  unit  what  we  call  ACE  box. 

The  physical  arrangement  of  the  whole  system  can  be  seen  in  Figure  1.  This  is  composed 
of  five  PC- 104  form  factor  cards  plus  a  metal  plane  with  a  lens  socket  on  the  top.  (The  size  of  a 
PC- 104  form  factor  card  is  the  same  as  the  size  of  a  VA  inches  floppy  disk.  The  lowest  three 
electronic  cards  of  the  rack  (the  power  supply,  the  Pentium  class  motherboard,  and  the  hard-drive 
module)  are  commercially  available  components.  The  power  supply  operates  from  any  battery  (or 
DC  power  source)  which  output  voltage  is  between  8V  and  25V.  The  hard  drive  module  is  needed 
for  standalone  operation.  In  case  of  standalone  operation  the  display  and  keyboard  are  not  even 
connected  to  the  ACE  box.  The  display  card  is  integrated  in  the  motherboard. 

The  upper  two  electronic  cards  are  the  DSP  module  and  the  platform.  The  DSP  module, 
which  controls  the  64x64  analogic  chip,  feeds  it  with  data,  and  reads  out  the  result  from  it.  The 
other  double-decker  board  is  the  platform.  The  main  functionality  of  the  platform  is  to  cany  the 
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64x64  analogic  chip,  and  provide  analog  and  digital  interfaces  between  the  analogic  chip  and  the 
DSP  module.  Above  the  platform,  there  is  a  metal  layer  which  has  only  mechanical  and  no 
electronic  functionality.  This  layer  holds  the  relay  lens  and  the  PAL  optics. 


Power  supply 


Figure  1.  The  physical  arrangement  of  the  ACE  box  (lower),  and  the  two  upper  modules. 


The  physical  arrangement  of  the  ACE  Box  with  the  ACE4k  Board  and  with  the  PAL  lens 
can  be  seen  in  Figure  2.  The  top  module  is  the  ACE4k  Platform.  The  C-mount  lens  socket  is 
opened  on  the  first  image. 
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Figure  2  Physical  arrangement  of  the  ACE4k  Box  with  the  ACE4k  Board  and  with  the  PAL 
lens.  The  ACE4k  Platform  with  the  ACE4k  chip  can  be  seen  on  the  top.  Here  the  rack 
contains  two  extra  cards,  a  framegrabber  and  a  network  card. 

Since  the  PAL  optic  creates  a  virtual  image,  which  is  inside  in  the  lens,  we  need  to  have  a 
relay  lens  system  to  project  a  real  image  to  the  surface  of  the  analogic  sensory  chip.  This  relay  lens 
optics  must  be  matched  with  the  physical  size  of  the  sensor.  The  PAL  lens  was  designed  to 
support  different  camera  input  as  well,  therefore  the  long  tube  is  necessary  to  project  sharp  images 
on  the  chip  surface. 


Detailed  Hardware  Description  of  the  ACE4k  Box 

As  it  was  described  in  the  previous  section  the  ACE4k  Box  is  composed  of  three  printed 
circuit  cards.  The  first  is  the  DSP  Module,  while  the  second  two  cards  constitute  the  ACE4k 
Platform  which  carries  the  ACE4k  chip.  These  components  are  implemented  on  PC-104  sized 
boards.  The  DSP  Module  and  the  ACE4k  Platform  are  connected  via  the  Platform  Bus. 
Description  of  these  modules  can  be  found  in  the  following  subsections. 

The  pictures  of  the  ACE4k  Platform  can  be  seen  in  Figure  3.  The  height  of  the  ACE4k 
Platform  and  the  DSP  Module  are  the  same  as  a  normal  PC/104plus  card. 
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PLD  Module  Analog  Module 


Figure  3  Pictures  of  the  ACE4k  Platform 

DSP  Module 

The  DSP  Module  contains  a  TMS320C6202  high  performance  DSP,  with  16MB  125MHz 
SDRAM,  concurrent  PCI  interface  and  a  Platform  Bus  which  is  32  bit  wide  data  and  16  bit 
address  bus  62.5MHz  asynchronous  CNN  Platform  Bus  interface.  This  DSP  controls  the  CNN 
chip  with  very  fine  (16-80ns)  timings. 

The  architecture  of  the  DSP  Module  can  be  seen  in  Figure  4.  It  is  a  relatively  simple 
construction  with  high-performance.  The  specialty  of  the  TMS320C6202  DSP  is  that  it  has  a  fast 
32  bit  wide  expansion  bus,  which  is  directly  interfaced  to  the  PCI  bridge  chip.  Via  this  expansion 
bus,  the  133  Mbyte/sec  PCI  communication  goes  through  the  DSP  and  reaches  its  target,  the 
SDRAM.  The  expansion  bus  transfers  data  to  the  memory  via  the  auxiliary  channel  of  the  DSP 
DMA  controller.  The  priority  of  this  DMA  channel  can  be  set  to  the  highest  level.  Therefore  the 
DMA  controller  has  an  internal  FIFO  to  store  temporally  the  data  during  the  memory  arbitration 
process.  The  DMA  controller  can  transfer  data  without  CPU  intervention. 
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Figure  4.  The  block  diagram  of  the  SPM6020  DSP  Module  architecture. 

The  memory  bus  of  the  DSP  has  three  times  bigger  data  bandwidth  than  the  PCI  bus.  The 
bandwidth  required  by  different  video  streams  is  less  than  10%  or  less  than  3%  in  RGB  or 
monochrome  cases  respectively.  Due  to  these  small  relations,  the  DSP  can  operate  close  to  its  full 
speed  even  in  case  of  continuos  RGB  image  flow. 

The  DSP  board  utilizes  most  of  the  features  of  the  TMS320C6202  high  performance  DSP 
on  a  very  small  PCI/104  form  factor.  The  board  features  are: 

•  TMS320C6202  based  PCI  104  bus  DSP  accelerator  board 

•  250  MHz  TMS320C6202  Processor-  4ns  cycle  time  -  2000  MIPS  operation 

•  16  MB  of  Synchronous  SDRAM,  500  MB/s 

•  2  MB  FLASH 

•  Watch  Dog  Timer 

•  On-Board  3.3V  and  1.8V  Generation 

•  3.3V/5V  PCI  Interface 

•  3.3V  Platform  Bus  for  easy  and  high  speed  interfacing  to  the  DSP  (32  bit  data  16  bit 
address  buffered  version  of  Asynchronous  Memory  interface  of  DSP  chip) 

•  PCI104  Form  Factor  (PCI  bus  only  plus  Platform  Bus  connector) 

•  SyncBus  for  synchronous  operation  with  RTD’s  high  performance  PC/104plus  Data 
Acquisition  boards  or  other  type  of  analog  interfaces. 
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•  Multichannel  Buffered  Serial  Ports  (Tl/El,  MV1P,  SCSA,  ST  bus,  AC97  SP1 
Compatible) 

The  DSP  Module  itself  is  a  DSP  accelerator  board  without  analog  interface.  It  can  be  used 
as  a  high  performance  coprocessor  in  the  PC/104plus  system.  The  Platform  Bus  (which  is  actually 
the  Asynchronous  interface  of  the  DSP)  can  be  used  as  an  analog  interface  for  easy  and  high¬ 
speed  data  transfer  device  between  the  DSP  and  the  ACE4k  Platform. 

Platform  Bus 

The  Platform  Bus  contains  the  32-bit  data  bus  of  the  DSP  and  the  16  lower  bits  of  the 
address  bus.  It  is  a  62.5MHz  asynchronous  bus.  It  also  provides  power  and  GND  for  the  platform 
also.  Physically  this  bus  is  similar  to  the  ISA  connector  of  the  PC/104  standard.  3.3V  signal  levels 
are  used  on  the  bus. 

The  Platform  Bus  is  used  to  write  and  read  the  ACE4k  chip,  and  to  program  (write)  its 
environment.  Writing  of  the  chip  is  used  for  two  purposes.  Sometimes  it  is  used  to  transfer  an 
image  to  the  chip,  which  is  a  4,  16,  64  kByte-sized  continuos  data  block.  But  in  many  cases  only 
some  bytes  are  sent  to  reprogram  the  chip,  but  this  transfer  should  be  fast  to  achieve  efficient 
control  speed. 

Platform  Bus  Signal  Description 

The  Platform  Bus  is  a  32  bit  wide  asynchronous  buffered  3.3V  bus  with  maximum 
62.5MHz  bus  frequency. 

The  Platform  Bus  signals  can  be  grouped: 

•  32  data  lines  (PDO..  31), 

•  16  address  lines  (PA2..  17), 

•  flow  control  lines  (PREL,  PWEL,  POEL,  PCE2L,) 

•  ready  signal  (PRDYH)  for  slow  I/O  devices  to  lengthen  the  bus  cycle, 

•  62.5MHz  clock  (PBCLK),  which  is  the  DSP’s  master  clock  divided  by  four, 

•  interrupt  lines  (PINT6H,  PINT7H)  to  interrupt  the  DSP  from  the  I/O  device,  the 
Interrupt  is  occurred  after  a  low-high  transition  on  the  interrupt  line. 

•  the  two-channel  DSP  timer  input/output  lines  (PTOUTO/PTOUT1  -  PT1NP0/PTINP1) 

•  3.3V  and±5V  ±12V  power 

•  ground  lines. 

ACE4k  Platform 

The  ACE4k  Platform  is  placed  directly  above  to  the  DSP  Module  in  the  PC/104plus  stack. 
It  is  necessary  to  be  placed  on  the  top  level  of  the  rack.  This  means,  that  the  PCI  bus  does  not  go 
through  this  level,  which  enables  to  use  those  areas  on  the  PCB.  On  the  other  hand,  in  case  of 
direct  optical  input  of  the  chip,  this  card  carries  a  C-mount  lens  system  which  projects  the  image 
to  the  chip  surface. 

The  ACE4k  Platform  is  dedicated  to  carry  the  ACE4k  chip  with  the  optics.  Physically  this 
unit  consists  of  two  PC- 104  form  factor  boards,  which  are  placed  on  the  top  of  the  PC- 104  rack, 
since  the  lens  system  can  be  mounted  only  on  the  top  of  the  upper  board.  Beside  the  ACE4k  chip, 
the  unit  constitutes  of  a  PLD,  a  buffer  for  driving  the  asynchronous  Platform  Bus  and  some  latches 
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for  storing  addresses  and  temporal  control  bits.  The  platform  board  contains  some  DC  to  DC 
converters  also  for  providing  stable  power  for  the  ACE4k  chip,  according  to  its  specification.  The 
block  diagram  of  the  platform  card  is  shown  in  Figure  5. 


Figure  5.  The  block  diagram  of  the  ACE4k  Platform. 

Because  the  ACE4k  chip  has  a  16  line  analog  bus  for  analog  I/O  transfer,  the  fine  analog 
lines  and  noisy  digital  signals  should  be  carefully  separated.  Therefore  the  ACE4k  Platform 
constitutes  of  two  PC- 104  form  factor  boards.  The  upper  board  hosts  the  ACE4k  chip  together 
with  analog  parts  (AD  and  DA  converters).  The  lower  board  contains  PLD  devices  to  control 
analog  transfer  between  the  ACE4k  chip  and  converters  and  to  provide  digital  interface  to  the  DSP 
Module. 

ACE4k  Platform  Analog  Module 

The  analog  module  contains  the  ACE4k  chip,  three  DA  converters  (two  of  them  for  signals 
and  one  for  reference  values),  and  four  AD  converters  with  their  OP  As. 

ACE4k  chip 

The  ACE4k  chip  can  process  binary  or  gray-scale  images  as  well.  The  chip  interface 
consists  two  bi-directional  data  buses  (one  digital  and  one  analog,  both  with  a  width  of  16-lines), 
and  a  control  bus  comprising  several  address  buses  and  5  control  signals  (a  chip  enable,  a  R/W 
signal,  a  control  signal  to  put  the  chip  in  either  programming  or  operation  mode,  and  a  global 
strobe).  All  data  interchange,  analog  and  digital,  are  suppose  to  work  at  a  maximum  rate  of  10 
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MHz.  The  total  CNN  array  size  is  64x64,  with  four  local  analog  and  binary  pixel  memory.  The 
analog  signal  level  is  in  the  range  of  0.6V- 1.4V.  The  power  supply  is  3.3V  with  worst  case  power 
dissipation  of  1 W.  The  package  of  the  chip  is  ceramic  pin  grid  array  (PGA  120). 

DA  converters 

The  AD8600  contains  16  independent  voltage  output  digital-to-analog  converters  that 
share  a  common  external  reference  input  voltage.  Each  DAC  has  its  own  DAC  register  and  input 
register  to  allow  double  buffering,  and  can  be  independently  addressed.  An  8-bit  parallel  data 
input,  four  address  pins,  a  chip  select,  load,  enable,  R/W,  and  RS  provide  the  digital  interface.  The 
analog  output  channels  have  full-scale  voltage  levels.  The  settling  time  of  the  outputs,  start  the 
settling  transient  at  the  same  time  allowed  by  the  double  buffering,  is  in  the  range  of  0.1 -0.5  |is. 
The  analog  output  signal  level  should  be  in  the  range  of  0.6V- 1.4V,  determined  by  two  reference 
voltages.  The  power  supply  is  ±5V  with  worst  case  power  dissipation  of  350  mW.  The  package  of 
the  chip  is  44-lead  plastic  lead  chip  carrier  (PLCC-44). 

AD  converters 

The  type  of  AD  converters  are  THS10G84.  There  are  four  ADs  and  each  AD  contains  four 
independent  channels  and  16  step  wide  FIFO.  The  resolution  is  10  bits.  This  provides  that  at  the 
same  time  all  the  16  analog  lines  of  the  ACE4k  chip  can  be  sampled  and  while  their  conversion  is 
processed  the  ACE4k  chip  can  put  the  next  data  to  its  lines.  The  ADs  are  connected  through 
operational  amplifiers  to  the  ACE4k  chip  therefore  level  shifting  and  calibration  is  possible  for 
fine  tuning. 


Results  of  PAL  image  processing  with  the  ACE  Box 

Two  major  groups  of  tests  were  evaluated.  The  first  one  used  the  direct  optical  input  of  the 
ACE4k  chip  through  the  PAF  lens.  The  second  test  series  was  made  on  image  sequences 
prerecorded  by  a  camera  using  PAF  lens.  These  two  major  groups  were  separated  because  the 
image  resolution  of  a  360  degrees  image  on  a  low  resolution  chip  is  quite  poor.  Therefore  larger 
sized  image  sequence  was  used  to  be  able  examine  image  processing  task  solved  on  the  chip 
avoiding  the  low  resolution.  The  targeted  problem  was  to  detect  movement  around  the  360 
degrees  and  some  basic  classification  tasks  were  mainly  concerned. 

Direct  on  chip  optical  input  experiments 

The  ACE4k  chip  has  no  dedicated  direct  optical  input.  Nevertheless,  the  CMOS  surface  of 
the  analog  image  memories  (LAMs)  gives  the  possibility  of  image  acquisition.  Figure  6  shows  a 
64x64  size  image  captured  by  the  chip.  The  relative  low  sensitivity  of  light  intensity  requires  a 
quite  long  integration  time.  The  accuracy  of  the  image  acquisition  is  between  7-8  bit. 
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Figure  6  Optical  acquisition  with  the  ACE4k  chip.  Image  accuracy  is  around  7-8  bit. 


The  next  snapshots  in  Figure  7  show  images  captured  by  the  direct  optical  input  of  the 
ACE4k  chip  through  the  PAL  lens. 


Figure  7  Consecutive  snapshots  of  direct  optical  input  of  the  ACE4k  chip  using  PAL  lens. 


The  low  resolution  makes  almost  impossible  to  recognize  objects  by  humans  either.  Since 
this  chip  was  not  designed  to  have  dedicated  direct  optical  input  the  light  sensitivity  of  the  chip  is 
quite  low.  It  means  that  without  a  strong  light  source  this  optical  input  can  not  be  used. 
Additionally  to  that,  noise  effects  have  strong  influence  on  images.  We  used  sophisticated 
methods  to  fdter  out  regular  and  stochastic  errors. 


Nevertheless,  some  basic  movement  detection  was  made  on  this  sequence.  This  is  shown 
in  Figure  8. 


The  movement  detection  was  based  on  mainly  temporal  homotopic  filtering,  gray-scale 
range  selection,  and  mathematical  morphology  operations.  These  operations  were  executed  mainly 
on  the  ACE4k  chip.  Measured  executing  time  of  this  algorithm  proves  that  fast  objects  can  be 
tracked  by  using  the  system.  More  detailed  evaluation  will  be  given  after  the  next  subsection.  Due 
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to  the  previously  mentioned  constraints,  namely  the  relatively  low  resolution  and  the  low  optical 
sensitivity  of  the  ACE4k  chip,  further  investigations  were  necessary.  These  measurements  are 
summarized  in  the  next  Section. 

PAL  camera  experiments 

In  these  experiments  the  optical  input  of  the  chip  was  not  used.  The  PAL  lens  was 
mounted  on  a  standard  video  camera  and  the  captured  images  were  processed  on  the  ACE4k  chip. 
This  makes  possible  to  test  processing  capabilities  of  the  ACE4k  chip  without  using  its  image 
capturing  capabilities.  Consecutive  snapshots  in  Figure  9  show  images  captured  by  the  camera 
through  the  PAL  lens,  meanwhile  Figure  10  shows  a  transformed  image  for  demonstration 
purpose.  The  processed  image  size  is  320x300.  The  reason  why  the  resolution  is  not  NxN  is 
because  we  did  not  have  a  square  pixel  camera.  However,  this  did  not  cause  any  problem  later  on. 


Figure  9  Consecutive  snapshots  of  a  video  flow  using  PAL  lens  optical  input.  Processed  image 
size  is  320x300. 


Figure  10  A  snapshot  of  a  video  flow  using  PAL  lens  optical  input  converted  from  the  panoramic 
view  into  a  stripe  image  for  demonstration  purpose  (note,  that  this  kind  of  time 
consuming  color  transformation  is  not  part  of  the  presented  algorithm). 


The  images  were  separated  to  small  chip  sized  blocks  and  these  blocks  were  processed 
sequentially  one  after  the  other  on  the  ACE4k  chip.  This  ability  is  incorporated  in  the  system  in  an 
automatic  way  so  any  size  of  image  sequences  can  be  processed  with  this  system.  The  speed  of  the 
ACE4k  chip  makes  possible  to  work  on  large  images  on  video  rate. 

The  special  projection  of  the  PAL  lens  makes  this  problem  far  from  trivial.  Depending  on 
the  distance  from  the  camera,  relative  sizes  of  objects  change  dramatically  from  small  patches  to 
almost  half  size  of  image  area.  Therefore,  sophisticated  methods  are  necessary  to  distinguish 
separate  objects  and  extract  important  features  of  them. 
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The  flowchart  of  the  movement  detection  and  classification  algorithm  is  shown  in 
Figure  11.  The  motion  detection  and  object  classification  algorithm  has  five  major  parts,  namely: 

I.  Optical  acquisition  and  background  estimation 

II.  Motion  map  generation  with  morphological  filtering 

III.  Pre-filtering  characteristic  feature  extraction 

IV.  Feature  based  classification  and  position  determination 

V.  Object  tracking  and  path  tracing 

The  first  part  of  the  algorithm  performs  background  estimation  continuously  from  the 
captured  image  sequence.  This  also  includes  an  extensive  pre-filtering  with  noise  filtering,  and 
temporal  filtering.  The  second  part  of  the  algorithm  generates  a  raw  motion  map.  This  part 
includes  spatial  morphological  and  wave  type  filtering  running  on  the  ACE4k  chip.  These  types  of 
operations  would  require  too  long  execution  time  on  a  DSP  therefore  the  ACE4k  chip  is  an  ideal 
tool  to  accomplish  these  tasks.  The  third  part  of  the  algorithm  makes  a  coordinate  transformation 
from  the  polar  representation  in  order  to  extract  characteristic  features  of  objects.  The  chip  also 
plays  important  role  here.  Statistical  features  are  extracted  on  the  DSP.  The  fourth  part  is  the 
classification  and  position  determination.  It  generates  several  true  and  false  object  candidates.  The 
fifth  part  filters  and  revises  the  classification  result  based  on  dynamical  class  models  and 
constructs  tracking  routes. 


EOARD  F61775-01-WE010 


12  AnaLogic  Ltd.  Budapest,  Hungary 


Interim  report:  Experimental  results  of  the  opto-electrical  image  acquisition  and  processing  unit  with  PAL  optics 


Object  path  database 

Figure  1 1  Flowchart  of  the  motion  detection  and  object  classification  algorithm. 
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The  time  consuming  of  the  algorithm  is  shown  in  the  next  table. 


Time  consuming  of  the  movement  ai 

ad  classificat 

ion  algorithm 

Fu 

nction 

Time 

Notation 

Image  acquisition 

50  ps 

on  chip  image  sensing 

Position  and  illumination  verification 

50ps 

3*Ttemplate+5*T|0gic 

Background  extraction  and  tracking  (with  image 
transfer) 

41  Ops 

with  DSP 

Intensity  change  detection 

20ps 

Morphological  and  wave-type  filtering 

60ps 

h-5 

HH 

Polar-Cartesian  transformation 

Resolution  correction  and  radii  directional  closing 

with  image  trasnfer 

Object  purging  by  horizontal  symmetry  checking 

50ps 

with  image  trasnfer 

Radii  directional  histogram  generation 

mm 

Histogram  refining 

IH 

> 

Curve  shape  based  classification 

loops 

6 

Tracking  and  path  finding 

lOOps 

Summary:  Chip 

280ps 

Summary:  DSP 

700ps 

Texas,  C6201,250MHz 

Total  time  1: 

980ps~lms 

chip  size  image 

Total  time  2: 

40ms 

large  image  handling  (320x300) 

As  a  conclusion,  a  CNN  chip  if  it  has  a  proper  direct  optical  input  could  solve  this  task 
with  around  1000  frame/sec  while  using  camera  with  large  images  video  rate  (25Hz)  can  be 
achieved.  The  chip  execution  time  is  valid  not  only  for  the  ACE4k  chip  but  also  larger  array  size 
chip  because  template  and  logic  operations  require  almost  the  same  execution  time.  The  execution 
time  of  the  DSP  related  operations  depend  strongly  on  the  chosen  DSP  architecture,  clock 
frequency  and  the  image  size. 
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Detailed  algorithm  description 

As  it  was  shown  in  the  previous  section,  the  algorithm  contains  five  major  steps.  Here  we 
give  detailed  description  of  these  steps.  The  first  step  of  the  algorithm  is  the  background 
estimation.  We  applied  a  spatial-temporal  non-linear  method  for  the  permanent  calculation  of  the 
actual  background.  This  method  relies  on  still  background  processing.  A  region  was  considered  to 
belong  to  the  background,  if  its  intensity/hue  values  stayed  within  given  intensity/hue  interval  for 
a  given  time.  The  size  of  the  intensity/hue  interval  was  determined  by  the  stochastic  noise  of  the 
camera  and  the  background.  Under  background  noise,  we  mean  those  minor  changes  of  the  scene, 
which  are  not  considered  objects  to  be  traced  (like  leafs  or  branches  propelled  by  the  wind).  The 
time  limit  was  derived  from  the  properties  of  the  tracked  object  (velocity,  distance,  etc).  We 
considered  a  tracked  object  to  be  disappeared,  if  there  were  no  movements  in  the  disappearing 
position  for  a  well-defined  time. 

The  second  step  of  the  algorithm  is  the  extraction  of  all  the  moving  objects.  In  a 
straightforward  manner,  we  extracted  those  regions  on  the  image,  which  were  changing  comparing 
to  the  permanently  updated  background.  Naturally,  the  raw  difference  map  is  noisy,  and  the 
objects  were  fragmented  (Figure  12b)  on  it.  The  second  part  of  this  step  is  responsible  for  the 
noise  filtering  and  object  reconstruction.  We  applied  morphological  filtering  (opening)  to  get  rid 
of  the  small  sized  objects,  which  are  considered  to  be  noises.  Then,  we  used  wave  type  operators 
to  reconnect  the  fragmented  body  parts.  Figure  12  shows  the  extraction  of  moving  parts  and  the 
outcome  of  the  filtering.  The  resulting  images  are  used  to  estimate  object  properties  in  the  next 
step. 


a)  b)  c) 

Figure  12  Moving  object  extraction,  a)  captured  image;  b)  extracted  moving  parts;  c) 
morphologically  filtered  moving  regions. 


The  third  step  of  the  algorithm  analyzes  the  objects  and  extracts  characteristic  features.  A 
special  profile  of  each  frame  is  constructed  from  the  resulting  image  of  the  previous  step  by 
applying  a  special  radial  projected  histogram  transformation.  Geometrically  this  transformation 
projects  a  vertical  half-plane  (the  edge  of  the  half  plane  is  the  vertical  axe  of  the  PAL  optics)  into  a 
single  scalar.  When  this  transformation  was  applied  to  the  result  of  the  previous  step,  we  got 
curves  shown  in  Figure  13.  After  having  these  curves,  we  applied  some  smoothing  and  refining  on 
them,  to  avoid  misclassification  in  the  next  step. 
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Figure  13  Typical  feature  curves  derived  from  the  radial  directional  histogram  a)  before  and  b) 
after  the  refining  process.  Horizontal  axis  shows  direction  in  degree  while  the  vertical 
axis  shows  the  number  of  black  pixels  which  is  related  to  the  object  height. 

In  the  fourth  step  we  made  the  classification.  We  used  the  above  described  special  shape 
representation  of  the  moving  objects  as  the  base  of  the  object  classification.  Depending  on  the 
height,  size,  and  other  properties  of  the  profile,  robust  classification  can  be  achieved.  This  method 
is  capable  to  distinguish  humans,  vehicles,  flying  objects,  etc.  We  used  the  following  object 
classes  and  dynamic  model  constraints  to  classify  the  objects. 


Class 

Feature 

Dynamic  constraint 

Standing  or  running  person 

Gaussian-like  shape  with  one  or 
more  close  peaks 

Relatively  slow  movement 

Moving  on  the  ground  plan 

Vehicle 

Wide  rectangular  shapes 

Relatively  fast  movement 

Moving  on  the  ground  plan 

Airplane  and  helicopters 

Wide  Gaussian-like  shape  with 
small  extension 

Even  very  fast  movement 

Moving  over  the  ground  plan  (above 
the  horizon) 

In  the  same  step,  after  the  identification  of  the  specific  profiles,  the  distance  of  each  object 
can  be  determined.  For  this  we  used  the  original  and  the  projected  physical  size  information.  In 
this  way  we  gained  a  database  (a  set  of  points)  consisting  a  series  of  raw  detected  events  without 
trajectory  consistency.  Figure  14  shows  the  raw  XYT  (two  spatial  and  one  temporal  dimension) 
object  data  for  the  sample  video  sequence. 
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Figure  14.  The  raw  XYT  object  database  without  additional  tracking  and  classification  process. 

The  diagram  shows  only  the  southern  half  plan,  because  in  our  example,  there  were  no 
movements  in  the  northern  half  plan. 

In  the  fifth  step  of  the  algorithm,  we  calculate  the  final  tracking  curves  of  the  objects,  by 
utilizing  consistency  and  kinematics  information.  At  the  same  time  we  can  get  rid  of  the  single 
point  misclassified  moving  objects,  which  are  not  constructing  continuos  tracks.  Object  paths  are 
separated  and  classified  based  on  dynamical  class  models.  Also  these  models  are  used  to  refine 
routes  (Figure  15). 


The  object  positions  -  directions,  distance,  and  type  -  can  be  superimposed  to  the  input 
sequence  in  real  time  (Figure  16). 


Figure  15  a)  The  database  was  filtered  and  classified  for  moving  persons;  b)  the  same  sample 
tracking  data  are  temporally  and  spatially  filtered  in  order  to  get  rid  of  the  detection 
noise. 
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Figure  16.  The  tracking  information  -  both  distance  and  position  -  is  superimposed  into  a 
snapshot  of  the  input  sequence. 

Conclusions 

We  proved  that  the  ACE4k  chip  with  a  companion  DSP  is  capable  to  solve  object 
classification  and  tracking  problem  real-time.  Due  to  the  lack  of  a  good  direct  optical  input  of  the 
ACE4k  chip,  we  had  to  use  a  camera  with  PAL  optics  as  the  input  source.  Time  consuming  of  this 
algorithm  makes  possible  to  track  even  very  fast  objects  in  the  scene,  although  further 
optimization  and  careful  system  design  is  necessary.  Hopefully,  the  ACE16K  chip  with  array  size 
128x128  and  with  direct  optical  input  will  overcome  this  problem. 


EOARD  F61775-01-WE010 


18  AnaLogic  Ltd.  Budapest,  Hungary 


Interim  report:  Experimental  results  of  the  opto-electrical  image  acquisition  and  processing  unit  with  PAL  optics 


Appendix  1:  Proposal  for  further  research  activities 

The  recent  CNN  chip  development  effort  led  to  a  128x128  sized  CNN  focal  plane  array 
processor  with  excellent  optical  input  characteristic.  This  chip,  called  ACE  16k,  is  already 
fabricated,  and  now  it  is  under  measurements.  The  higher  resolution  of  this  chip,  and  the  excellent 
optical  characteristic  suggest  the  idea  to  combine  this  chip  with  PAL  optics.  In  the  spirit  of  this, 
here  we  propose  three  possible  research  activities  in  the  framework  of  a  next  EOARD  contract: 

1.  Developing  a  standalone  device,  which  contains  a  PAL  optics,  a  128x128  CNN  chip,  a 
DSP,  and  a  communication  unit,  and  which  implements  the  tracking/tracing  algorithm 
develop  in  the  EOARD  F61775-01-WE010.  The  estimated  frame-rate  of  the  unit  will 
be  around  1000  FPS. 

2.  Developing  a  standalone  device,  which  contains  a  HUMANOID  PAL  optics,  a 
128x128  CNN  chips,  a  DSP,  and  a  communication  unit,  and  which  implements  a 
modified  version  of  the  tracking/tracing  algorithm  develop  in  the  EOARD  F61 775-01- 
WE010.  The  modified  version  could  control  the  fovea  of  the  HUMANOID  PAL  optics 
to  follow  a  selected  target.  The  estimated  frame-rate  of  the  unit  will  be  around  1000 
FPS. 

3.  Create  an  advanced  version  of  the  system  what  we  developed  in  the  framework  of 
EOARD  F61775-01-WE010  which  could  recognize  and  trace  missile  launches  real¬ 
time,  even  if  there  were  a  large  number  of  missiles  launched  at  the  same  time. 


These  research  activities  would  be  pursued  by  AnaLogic  Computers  Ltd.  alone.  In  all 
follow-on  projects,  the  delivery  of  an  operational  prototype  for  test  purposes  is  negotiable. 


Appendix2:  List  of  the  video  demonstrations 

List  of  the  video  demonstrations: 


Chip_lnput.avi 
Chip_Motion.avi 
C  ameralnput.  avi 


:Chip  optical  image  acquisition 
:Motion  detection  using  chip  optic 
: Camera  input  image  series 
Camera_PreProc.avi  : Camera  input,  raw  motion  map 

Camera_Tracking.avi  :Camera  tracking  result  with  object  direction  and  distance 
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